fix(grpo): clip the KL log-ratio before exp to avoid inf overflow - #6637

The `kl_log_ratio_clip` added for issue #3015 clamped the K3 estimator
on both sides and promised to keep the KL term finite for any positive
value. Neither held.

Clamping the negative side changes a correct number. `exp` overflows
only for large positive input; a large negative log-ratio underflows
to zero and leaves K3 finite, growing as `-kl_log_ratio - 1`. At
`kl_log_ratio = -50` with `kl_log_ratio_clip = 10` the estimator
dropped from 49.0 to 9.0, with no error and no warning. The clamp is
now `max=` only.

The finiteness promise failed above the working dtype's exp
ceiling. float32 overflows at 88.7229, so `kl_log_ratio_clip=90.0`
clamped to 90 and still returned `inf`. That ceiling varies by
dtype: roughly 88.7 for float32 and bfloat16, but only 11.1 for
float16. Deriving it as `math.log(finfo.max)` is itself unsafe,
because the float64 result rounds up when cast back down and the
bound then overflows anyway. The clip is instead exponentiated as a
scalar in the tensor's own dtype, and a value that overflows raises
`ValueError` rather than silently returning `inf`.

The regression test never called the trainer. It reimplemented K3
in a local helper and asserted against that helper, so deleting the
clamp from all three trainers left it at 2 passed, exit 0. Four cases
now drive `_compute_loss` with hand-built inputs, one per property:
overflow tamed by a usable clip, an over-large clip rejected,
a large negative log-ratio left intact, and a non-binding clip as
a no-op. The dtype ceiling each case needs is read from the tensor
rather than written as a literal, so the same test is correct under
bfloat16 autocast and float16.

Three mutants confirm the tests bite: deleting the guard fails 2 of
the 4, restoring the two-sided clamp fails 2, and removing only the
`ValueError` fails 1. The clamp block stays byte-identical across grpo,
gmpo and gspo_token.

…low-v2

…ts value

A plain clamp on the log-ratio zeroes the K3 slope for every clipped token. With the bias correction
on (the default) the KL term then reduces to `K3(clip) * ratio`, and the ratio's gradient rewards a
lower policy log-prob: minimizing the loss pushes the policy further from the reference. Measured with
plain torch at `x = 20`, `clip = 10`: the gradient with respect to the policy log-prob is `+2.2e4`
under the clamp, `-10` with the clip straight-through, and `-32` unclipped in float32, where the true
`-x * ratio = -20` is already lost to cancellation. The clip now keeps its value but passes the
gradient, in all three copies of the block, so a clipped token still pulls the policy toward the
reference with the slope at the clip.

`GRPOConfig` accepted any float: a non-positive clip invents KL at an exact policy/reference match, and
`-inf` reaches the estimator as `inf` past the trainer's overflow guard, which only rejects values whose
exponential overflows. `__post_init__` now requires a positive finite number. The docstring and help
said a policy drifting far above the reference overflows; a large positive `log(pi_ref / pi_theta)`
means the policy sits far below it.

Tests: the gradient of the loss with respect to a clipped token's log-prob must be negative, with the
bias correction on and off; the KL term with zero advantages must stay finite and positive; five
non-positive or non-finite clips must be rejected.